Containerized CI + Cleanup testing #218
Containerized CI + Cleanup testing #218clark-evans merged 88 commits intoufs-community:gsl/developfrom
Conversation
|
@AndersJensen-NOAA @joeolson42 Can you guys review this PR? |
joeolson42
left a comment
joeolson42
left a comment
There was a problem hiding this comment.
This is a very slim perusal more than a strict review. Getting this PR committed with bugs is probably not a set back relative to where we are today with the server issues.
clark-evans
left a comment
clark-evans
left a comment
There was a problem hiding this comment.
General comment: while I like that each test has its own directory with its own namelist(s), stream_lists, and streams files, there is a lot of duplication between them. Would it be cleaner to place common files into a common directory -- or perhaps two common directories, one for MPAS-Dev and one for ufs-community tests -- from which each set of tests could pull as needed?
I like this idea, but I don't have time to address it anytime soon. |
That's fine with me. Some notes for you, me, or whoever else might do this later:
|
|
@clark-evans @ligiabernardet I wasn't able to get the image to push to ghcr/ufs-community. I'm sure it's something permission related, but I don't have the time to chase it down. Is it okay if we use my Dockerhub account and I open an issue to revisit this later? |
That's fine with me. I'm not worried about safety in this context - just wanting to make sure all is well if something were to happen to you! Want me to open the issue, or are you OK with doing so? |
|
It sounds like a good approach for now. If we need to revisit anything regarding permissions in the repo, please reach out. |
clark-evans
left a comment
clark-evans
left a comment
There was a problem hiding this comment.
OK with me once my versioning comment is addressed.
5 participants
This PR updates to CI based testing system in several ways.
Testing cleanup
1a) Remove dependency of testing repository (https://github.com/barlage/mpas_testcase) from CI workflow
1b) Added namelist/stream case files to repository under testing_and_setup/ufs-community/cases. Previously these were in testing repository.
1c) Remove dependency on THREDDS server from CI workflow.
1d) Create GitHub Release assets with files from THREDDS server (e.g., physics tables and MPAS data files).
1e) New script to fetch GitHub release assets (testing_and_setup/ufs-community/data/get_data.sh)
Images for Containerized CI
2a) Added Dockerfiles for minimal images (e.g., GNU, Intel oneapi, nvhpc)
2b) Added Dockerfiles for dependencies (e.g., netCDF, and PnetCDF)
2c) Added new CI script that creates images from Dockerfiles and pushes to Dockerhub.
*We have MPAS images for GNU, Intel oneAPI, and NVHPCS, but only GNU has been implemented in this PR (Intel oneAPI will be part of a subsequent PR)
Containerized CI
3a) Updated CI script to run in container using images from Dockerhub (Created in 2c).
3b) Modify workflows to use namelists/streams in testing_and_setup/ufs-community/cases.
3c) Modify workflow scripts to fetch GitHub assets (Using 1e)
Comments
All existing CI tests pass*.
*NOTE. The MPAS-Dev tests are showing differences in the HEAD of gsl/develop, which we also see in this feature branch.
To download the release assets for running the regression tests offline;
./testing_and_setup/ufs-community/data/get_data.sh
The CI script to upload the images to Dockerhub wont work on a pull_request trigger, since it relies on Github secrets in the target repository. (This is default behavior for security concerns)
I updated this repo with the appropriate secrets, so once this PR is merged, the image building script will work.
Priority Reviewers